13. Demo 1: Data Cleaning and Preprocessing

Cd13652 C6 L1 DEMO 1 Part 1 V2

Data Cleaning and Pre-Processing Demo

Overview

Learn the fundamentals of data cleaning and pre-processing using the California Housing Dataset from the 1990 census. This demonstration walks through essential steps to prepare data for analysis.

Key Steps in Data Cleaning:

  • Initial Data Familiarization:

    • Review dataset documentation.
    • Identify target variables and potential features.

  • Data Inspection:

    • Use shape to check data dimensions.
    • Utilize head, tail, and describe methods for data overview.
    • Inspect data types with dtypes or info.

  • Handling Duplicates:

    • Detect duplicates using duplicated and remove them with drop_duplicates.

  • String to Number Conversion:

    • Transform currency values by removing non-numeric characters.
    • Convert strings to numerical types with astype.

  • Dealing with Non-Numeric Rows:

    • Identify and eliminate non-numeric data with Boolean masking.
    • Ensure numerical data casting for proper analysis.

  • Column Evaluation and Removal:

    • Assess the relevance of columns like 'households'.
    • Remove unnecessary columns using drop.

What's Next:

Prepare for more advanced cleaning techniques, including outlier detection and handling missing values.

Cd13652 C6 L1 DEMO 1 Part 2 V2

Further Data Cleaning and Handling Outliers

Introduction to Data Handling

  • Dataframe Composition: The dataset contains nine columns. Columns can be listed with attributes like columns.
  • Missing Values Identification: Use isna or the alternative isnull to identify missing data.

Handling Missing Data

  • Detecting NaNs: total_bedrooms column has 207 missing values, a small data percentage.
  • Solutions for Missing Data:
    • Direct Dropping: Not ideal due to data quantity loss.
    • Imputation: Using fillna to fill missing values with median values or employing SimpleImputer from scikit-learn for advanced operations.
  • Important Note on Data Integrity: Address imputation post data splitting to avoid data leakage.

Managing Outliers

  • Outlier Verification: Confirm that outliers are genuine data points.
  • Identification Methods: Box plots, facilitated by libraries such as Matplotlib and Seaborn, help visualize interquartile ranges to highlight outliers.
  • Specific Observations: Median house value shows capped data, suggesting possible dataset adjustment.

Next Concept